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Why GAO Did This Study 

In addition to providing health care 
to veterans, the Department of 
Veterans Affairs (VA) funds research 
that focuses on health conditions 
veterans may experience. According 
to VA, experts estimate that up to 
20 percent of Operation Enduring 
Freedom and Operation Iraqi 
Freedom veterans have experienced 
post-traumatic stress disorder 
(PTSD) and demand for PTSD 
treatment is increasing. Because of 
the importance of research in 
improving the services that veterans 
receive, GAO was asked to report on 
VA's funding of PTSD research, and 
its processes for funding PTSD 
research proposals, reviewing and 
incorporating research outcomes into 
clinical practice guidelines (CPG) — 
tools that offer clinicians 
recommendations for clinical 
services but do not require clinicians 
to provide one service over another — 
and determining which PTSD 
services are required to be made 
available at VA facilities. To do this 
work, GAO obtained and summarized 
VA data on the funding of PTSD 
research from its medical and 
prosthetic research appropriation 
through its intramural research 
program. GAO also reviewed relevant 
VA documents, such as those for 
developing CPGs and those related to 
VA's 2008 Uniform Mental Health 
Services in VA Medical Centers and 
Clinics handbook (Handbook), 
which defines certain mental health 
services that must be made available 
at VA facilities. GAO also interviewed 
VA officials. 
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What GAO Found 

Based on VA data GAO obtained and summarized, GAO found that the amount of 
funding VA provided for intramural PTSD research increased from $9.9 million in 
fiscal year 2005 to $24.5 million in fiscal year 2009. From fiscal year 2005 through 
fiscal year 2009, intramural PTSD research funding ranged from 2.5 percent to 
4.8 percent of VA's medical and prosthetic research appropriation. In addition, the 
number of PTSD research studies VA funded through the Merit Review Program 
and the Cooperative Studies Program (CSP) — VA's two primary funding 
mechanisms in its intramural research program — increased from 47 in fiscal year 
2005 to 96 in fiscal year 2009. 

According to VA officials, intramural research proposals, including those on 
PTSD, are funded primarily according to scientific merit in both the Merit Review 
Program and CSP. Proposals are evaluated by a panel of reviewers and scored 
based on their scientific merit. Directors of VA's research and development 
services — offices that focus on different research areas and administer VA's 
intramural research program — fund proposals based on their scores, typically up 
to a specified percentile. The number of proposals funded may vary based on 
budgetary considerations and, for a small number of proposals, responsiveness to 
VA research priority areas. 

VA has a process to review and incorporate relevant research outcomes to 
develop CPGs for a number of topics, including PTSD. VA relies on the policies of 
a joint VA and Department of Defense (DOD) work group — comprised of VA and 
DOD officials — to ensure that systematic reviews of relevant research outcomes 
are conducted when issuing CPGs. In brief, a systematic review is conducted to 
identify the most methodologically rigorous research studies that are applicable 
to each clinical question contained in the CPG. A group of subject matter experts 
then assesses the individual research studies in order to determine the overall 
quality of evidence available for each particular clinical question, considers the 
potential benefits and harms of a clinical intervention to determine its net effect, 
and, based on an assessment of the overall quality of the evidence and the net 
effect of an intervention, develops recommendations for the CPG. 

According to VA officials, the decision to require that two PTSD services — 
cognitive processing therapy and prolonged exposure therapy — be made available 
at VA facilities by including them in the Handbook was based on a review of 
research outcomes and the availability of existing resources. Specifically, VA 
officials told GAO that these two services were strongly recommended in the 2004 
PTSD CPG and had greater evidence supporting their effectiveness than other 
PTSD services. VA also told GAO that prior to the Handbook's 2008 issuance, VA 
had already begun investing resources in training programs for cognitive 
processing therapy in 2006 and prolonged exposure therapy in 2007. While VA 
provided some documentation regarding the decision-making process for PTSD 
services, VA officials explained that clinical decision-making processes are not 
typically expected to be documented in a formal manner. VA officials told GAO 
that they are currently clarifying language in the Handbook but do not plan to 
revise any requirements relating to PTSD services at this time. 

VA provided technical comments that GAO incorporated as appropriate. 



United States Government Accountability Office 



Contents 



Letter 



Background 

PTSD Research Funding Increased from Fiscal Year 2005 through 

Fiscal Year 2009 
VA Intramural PTSD Research Studies Are Funded Primarily 

According to Scientific Merit 
VA Has Established a Standardized Process to Review All Relevant 

PTSD Research Outcomes and Incorporate Them into a PTSD 

CPG 

VA Reviewed Research Outcomes and Used Existing Resources in 
Determining Which PTSD Services to Require Its Facilities to 
Provide or Make Available to Veterans 

Agency Comments 



1 

7 

10 
14 

20 



22 
24 



Appendix I 



VA Research Centers and Programs That Conduct 
or Support PTSD Research 



26 



Appendix II 



The Review Process for VA's Merit Review Program 28 



Appendix III 



The Review Process for VA's Cooperative Studies 
Program 31 



Appendix IV 



The Process for Developing VA/DOD Evidence-Based 
Clinical Practice Guidelines 34 



Appendix V 



GAO Contact and Staff Acknowledgments 



39 



Related GAO Products 



40 



Page i 



GAO-11-32 Veterans Affairs Post-Traumatic Stress Disorder Research 



Tables 



Table 1: The Office of Research and Development's Research and 

Development Services 7 

Table 2: Department of Veterans Affairs (VA) Intramural Research 
Study Funding Limits and Maximum Length of Award by 
Research and Development Service 8 

Table 3: Department of Veterans Affairs (VA) Intramural Post- 
Traumatic Stress Disorder (PTSD) Research Funding and 
VA's Medical and Prosthetic Research Appropriation, 
Fiscal Years 2005 through 2009 13 

Table 4: Scientific Merit Scoring Criteria in the Department of 

Veterans Affairs' (VA) Merit Review Program 16 

Table 5: Overview of Funding Recommendations Provided by the 
Department of Veterans Affairs' (VA) Cooperative Studies 
Scientific Merit Review Board 19 

Table 6: Department of Veterans Affairs (VA) Research Centers and 
Programs That Conduct or Support Post-Traumatic Stress 
Disorder (PTSD) Research 26 

Table 7: Scoring Guidelines for the Department of Veterans Affairs' 

(VA) Merit Review Program 30 

Table 8: Categorization of Ratings Used by the Guideline Working 
Group Established by the Department of Veterans Affairs 
(VA)/Department of Defense (DOD) Evidence-Based 
Practice Work Group to Identify the Level of Evidence of 
Research Outcomes 36 

Table 9: Categorization of Ratings Used by the Guideline Working 
Group Established by the Department of Veterans Affairs 
(VA)/Department of Defense (DOD) Evidence-Based 
Practice Work Group to Determine the Overall Quality of 
Research 37 

Table 10: Categorization of Ratings Used by the Guideline Working 
Group Established by the Department of Veterans Affairs 
(VA)/Department of Defense (DOD) Evidence-Based 
Practice Work Group to Determine the Net Effect of the 
Intervention as Described in the Research 38 

Table 11: Grade of Evidence-Based Recommendation Assigned by 
the Guideline Working Group Established by the 
Department of Veterans Affairs (VA)/Department of 
Defense (DOD) Evidence-Based Practice Work Group 
Based on the Overall Quality of Evidence and the Net 
Effect of the Intervention as Described in the Research 38 



Page ii 



GAO-11-32 Veterans Affairs Post-Traumatic Stress Disorder Research 



Figures 



Figure 1: Department of Veterans Affairs (VA) Intramural Funding 
for Post-Traumatic Stress Disorder (PTSD) Research 
Studies and Other PTSD Research-Related Funding, Fiscal 
Years 2005 through 2009 1 1 

Figure 2: Department of Veterans Affairs (VA) Intramural Post- 
Traumatic Stress Disorder (PTSD) Funding for Merit 
Review Program Studies and Cooperative Studies 
Program (CSP) Studies, Fiscal Years 2005 through 2009 12 

Figure 3: Number of Department of Veterans Affairs (VA) 

Intramural Post-Traumatic Stress Disorder (PTSD) 

Research Studies Funded per Fiscal Year, Fiscal Years 

2005 through 2009 14 

Figure 4: Overview of Submission, Review, and Funding Process in 
Department of Veterans Affairs' (VA) Merit Review 
Program 15 

Figure 5: Overview of Submission, Review, and Funding for a 

Department of Veterans Affairs' (VA) Cooperative Studies 
Program (CSP) Proposal 18 

Figure 6: Summary of the Department of Veterans Affairs' (VA) 

Merit Review Process for the Merit Review Program 29 

Figure 7: Summary of the Department of Veterans Affairs' (VA) 

Cooperative Studies Program (CSP) Review Process 32 

Figure 8: Overview of the Process Used by the Guideline Working 
Group Established by the Department of Veterans Affairs 
(VA)/Department of Defense (DOD) Evidence-Based 
Practice Work Group to Grade Evidence-Based 
Recommendations for Clinical Practice Guidelines 36 



Page ili 



GAO-11-32 Veterans Affairs Post-Traumatic Stress Disorder Research 



Abbreviations 



CPG clinical practice guideline 

CSP Cooperative Studies Program 

DOD Department of Defense 

Handbook Uniform Mental Health Services in VA Medical Centers 

and Clinics handbook 

ORD Office of Research and Development 

PTSD post-traumatic stress disorder 

VA Department of Veterans Affairs 



This is a work of the U.S. government and is not subject to copyright protection in the 
United States. The published product may be reproduced and distributed in its entirety 
without further permission from GAO. However, because this work may contain 
copyrighted images or other material, permission from the copyright holder may be 
necessary if you wish to reproduce this material separately. 



Page iv 



GAO-11-32 Veterans Affairs Post-Traumatic Stress Disorder Research 



GAP 

Accountability * Integrity * Reliability 



United States Government Accountability Office 
Washington, DC 20548 



January 24, 2011 

The Honorable Michael H. Michaud 
Ranking Member 
Subcommittee on Health 
Committee on Veterans' Affairs 
House of Representatives 

Dear Mr. Michaud: 

In addition to providing health care to about 5 million veterans each year, 
the Department of Veterans Affairs (VA) also funds research that focuses 
on the specific health conditions that veterans may experience. One 
condition that is examined in VA-funded research is post-traumatic stress 
disorder (PTSD), an anxiety disorder that can occur after a person is 
exposed to a life-threatening event. 1 Veterans diagnosed with PTSD may 
experience problems sleeping, maintaining relationships, and returning to 
their previous civilian lives. 2 According to VA, experts estimate that up to 
20 percent of Operation Enduring Freedom and Operation Iraqi Freedom 
veterans, 3 up to 10 percent of Gulf War veterans, and up to 30 percent of 
Vietnam War veterans have experienced PTSD. 4 Consequently, demand for 
PTSD treatment continues to grow. VA data show that from fiscal year 
2004 through fiscal year 2008, the number of unique veterans receiving 
treatment for PTSD increased by 60 percent from over 274,000 to over 
442,000. In particular, the number of Operation Iraqi Freedom and 
Operation Enduring Freedom veterans who received VA treatment for 



'American Psychiatric Association, Diagnostic and Statistical Manual of Mental 
Disorders, Fourth Edition, Text Revision (Washington, D.C., 2000). 

2 Those diagnosed with PTSD may also suffer from other ailments, such as depression and 
substance abuse. 

J Operation Enduring Freedom, which began in October 2001, supports combat operations 
in Afghanistan and other locations, and Operation Iraqi Freedom, which began in March 
2003, supports combat operations in Iraq and other locations. In September 2010, 
Operation Iraqi Freedom became known as Operation New Dawn. 

'Estimates for veterans who have experienced PTSD vary. For example, according to VA, 
regarding Operation Enduring Freedom and Operation Iraqi Freedom veterans, one 2008 
RAND study found that approximately 14 percent of these veterans have experienced 
PTSD, while another 2004 study by Hoge et al. estimated that from 6 to 20 percent of those 
veterans have experienced PTSD. 
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PTSD increased from nearly 4,400 in fiscal year 2004 to over 69,000 in 
fiscal year 2008. 

Veterans diagnosed with PTSD can receive a range of treatments to 
manage their symptoms, including individual and group therapy and 
medication. Veterans can receive PTSD treatment on an outpatient basis at 
VA facilities such as medical centers, community-based outpatient clinics, 
and Vet Centers. 5 Veterans can also receive intensive treatment through 
VA medical center inpatient settings for acute care needs and through 
residential rehabilitation treatment programs for more prolonged 
rehabilitative care. 

VA has guidance and policies in place related to PTSD services. For 
example, in 2004, the joint VA/Department of Defense (DOD) Evidence- 
Based Practice Work Group 6 issued a CPG for PTSD. 7 The PTSD CPG is an 
educational tool for clinicians that provides evidence-based 
recommendations 8 for PTSD services based on a review of PTSD research 
outcomes. It does not require clinicians to provide one service over 
another. However, in a handbook issued in 2008 — Uniform Mental Health 
Services in VA Medical Centers and Clinics (Handbook) — VA for the first 
time required certain mental health services to be made available to 



"Vet Centers offer readjustment counseling — a wide range of psychosocial services that 
includes individual and group counseling and screening and referrals for medical issues — 
to eligible veterans and their families. 

6 Formed in 1999 and composed of VA and DOD officials, the VA/DOD Evidence-Based 
Practice Work Group makes decisions about which clinical practice guidelines (CPG) for 
specific conditions will be developed and oversees their development. As of August 2010, 
the VA/DOD Evidence-Based Practice Work Group had issued 24 CPGs on conditions such 
as chronic heart failure and major depressive disorder. Since VA and DOD issue the CPGs 
in a joint effort, they are intended to be used by both VA and DOD clinicians. The VA/DOD 
Evidence-Based Practice Work Group is supported by VA's Office of Quality and 
Performance and the U.S. Army Medical Department's Office of Quality Management. 

7 The PTSD CPG is formally known as the VA/DOD Clinical Practice Guideline for the 
Management of Post-Traumatic Stress (January 2004). 

8 Evidence-based care refers to approaches that have consistently been shown in controlled 
research to be effective for a particular condition or conditions. 
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veterans throughout the system. 9 For PTSD, the Handbook requires that 
cognitive processing therapy or prolonged exposure therapy, two 
evidence-based psychotherapies, be provided or made available at VA 
facilities. 10 

Through its intramural research program, VA funds studies on different 
topics, such as PTSD. VA intramural studies are conducted by VA 
investigators — that is, researchers who conduct VA research studies. VA 
intramural research refers to research that is funded by and conducted 
within VA. In fiscal year 2009, VA received an appropriation of $510 million 
for medical and prosthetic research. 11 This appropriation funds VA's 
intramural research program, including providing funding for necessary 
equipment and supplies. 12 VA also receives three medical care 
appropriations that support VA's intramural research by paying some of 
the costs associated with this research, for example, the salaries of VA 



'Veterans Health Administration Handbook 1160.01, Uniform Mental Health Services in 
VA Medical Centers and Clinics (Sept. 11, 2008). According to the Handbook, all veterans 
with PTSD must have access to cognitive processing therapy or prolonged exposure 
therapy. VA medical centers and very large community-based outpatient clinics (serving 
more than 10,000 unique veterans each year) must be able to provide staff to administer 
such services at the facilities. Large (serving 5,000 to 10,000 unique veterans each year) and 
midsized (serving 1,500 to 5,000 unique veterans each year) community-based outpatient 
clinics may provide these services through telemental health or contract care when 
necessary. In March 2010, VA issued the Veterans Health Administration Handbook 
1160.03, Programs for Veterans with Post-Traumatic Stress Disorder (PTSD), to provide 
additional information regarding the implementation of the PTSD requirements contained 
in the 2008 Handbook. 

10 Psychotherapies focus on changing individuals' behaviors, thoughts, perceptions, and 
emotions. Cognitive processing therapy utilizes trauma-specific cognitive techniques to 
help patients move past trauma-related thoughts and progress toward recovery. Prolonged 
exposure therapy works by helping individuals approach trauma-related thoughts, feelings, 
and situations that they have been avoiding because of the distress they cause. Repeated 
exposure to these thoughts, feelings, and situations can assist with reducing the likelihood 
they will cause distress. 

"VA's medical and prosthetic research appropriation pays for necessary expenses in 
carrying out programs of medical and prosthetic research and development as authorized 
by chapter 73 of title 38, United States Code, the authorizing laws for the Veterans Health 
Administration. See Pub. L. No. 110-329, 122 Stat. 3574, 3706 (2008). 

12 The medical and prosthetic research appropriation funds research on different topics. It is 
not limited to funding research on PTSD. 
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investigators who are also VA clinicians. 13 According to VA, in fiscal year 
2009, VA's medical care appropriations totaled about $41 billion and VA 
provided $433 million of that amount to support all research conducted at 
VA facilities. 14 

Because of the growing demand for PTSD services and the importance of 
research in improving the health care services that veterans receive, you 
asked us to report on VA's funding of PTSD research, VA's processes for 
reviewing PTSD research proposals, VA's incorporation of research 
outcomes into clinical practice, and VA's process for determining which 
PTSD services it requires VA facilities to provide or make available. This 
report will describe 

• how much funding VA provided for intramural PTSD research from its 
medical and prosthetic research appropriations from fiscal year 2005 
through fiscal year 2009, 

• how VA determines which intramural PTSD research studies will be 
funded, 

• how VA reviews PTSD research outcomes and incorporates them into 
its PTSD CPG, and 

• how VA determines which PTSD services it requires VA facilities to 
provide or make available. 

To describe how much funding VA provided for PTSD research from its 
medical and prosthetic research appropriations through its intramural 
research program from fiscal year 2005 through fiscal year 2009, we 
obtained and summarized data provided by VA's Office of Research and 



"VA's three medical care appropriations are (1) medical services, which provides funds for 
the provision of veterans' health care services; (2) medical support and compliance, which 
provides funds for expenses related to the administration of veterans' health care services; 
and (3) medical facilities, which provides funds for the operation and maintenance of VA's 
health care facilities. Each of these appropriations provides support to all research 
conducted at VA facilities. For example, the medical care appropriations pay for the 
salaries of VA investigators who are VA clinicians (according to VA, about 70 percent of VA 
investigators are VA clinicians), administrative costs such as those for payroll and human 
resources, and logistical and infrastructure costs. 

14 VA could not provide an estimate for the amount of funding VA spent from the medical 
care appropriations for intramural PTSD research. Of the $433 million, more than half went 
to personnel costs, including the salaries of clinicians and other VA facility staff. 
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Development (ORD), the office that manages the Merit Review Program 
and the Cooperative Studies Program (CSP) — the two primary funding 
mechanisms within VA's intramural research program. 15 VA officials define 
PTSD research as "the effort to acquire generalizable knowledge about 
causes, epidemiology, susceptibility and genetics, resilience, 
pathophysiology, prevention and treatment of post-traumatic stress 
disorder." Based on these data, for fiscal years 2005 through 2009, we 
calculated totals and trends for (1) overall intramural PTSD research 
funding — including PTSD research studies and other PTSD research- 
related funding, such as career development awards provided to junior VA 
investigators to conduct PTSD studies, salaries for VA investigators who 
are not VA clinicians, 16 funding for PTSD research conducted within ORD 
research centers, and PTSD research meetings, and (2) the number and 
type of intramural PTSD research studies funded. According to VA, the 
funding data we obtained from VA do not include funds provided to 
conduct or support intramural PTSD research from VA's medical care 
appropriations. 17 Because of this, they do not necessarily represent all the 
funding VA provided for PTSD research during this time. In addition, to 
obtain information about how VA funds research conducted at VA, we 
interviewed officials from VA's Veterans Health Administration's Office of 
Finance, ORD, and Veterans Integrated Service Networks. 18 We also 
reviewed VA documents regarding its total medical and prosthetic 
research appropriations. We tested the internal consistency and reliability 
of the PTSD research data by reviewing the data for obvious outliers and 



15 The data provided by ORD reflect the funding amounts VA investigators were authorized 
to use, not necessarily the amounts that were spent. VA officials said that the authorized 
amounts are very similar to the amounts that are spent. 

16 Salaries for VA investigators who are VA clinicians are paid from the medical services 
appropriation, one of the medical care appropriations. 

17 For example, VA's Veterans Integrated Service Networks, which receive funding from the 
medical care appropriations and include VA's medical facilities, may also fund research by 
VA investigators, but this research funding is not considered part of VA's intramural 
research program. According to ORD officials, about 95 percent of the PTSD research 
funded by VA is funded by VA's intramural research program. 

18 The management of VA's medical facilities is decentralized to 21 regional networks 
referred to as Veterans Integrated Service Networks. 
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performing consistency checks. We found the data to be sufficiently 
reliable for the purposes of this report. 19 

To describe how VA determines which intramural PTSD research studies 
will be funded, we interviewed VA officials responsible for managing VA's 
intramural research program. We also obtained and analyzed VA 
documents that describe VA's policies for submitting, reviewing, and 
funding intramural research proposals. In addition, to understand the 
process VA uses to evaluate intramural research proposals, we observed a 
meeting in which research proposals were evaluated. 

To describe how VA reviews PTSD research outcomes and incorporates 
them into its PTSD CPG to inform clinical practice at VA, we interviewed 
VA officials who are responsible for developing CPGs. We also obtained 
and reviewed the PTSD CPG and documents related to CPG development. 

To describe how VA determines which PTSD services it requires VA 
facilities to provide or make available to veterans, we interviewed VA 
officials responsible for determining these requirements. We also obtained 
and reviewed documents that describe the PTSD service requirements. 

We conducted this performance audit from May 2010 through November 
2010 in accordance with generally accepted government auditing 
standards. Those standards require that we plan and perform the audit to 
obtain sufficient, appropriate evidence to provide a reasonable basis for 
our findings and conclusions based on our audit objectives. We believe 
that the evidence obtained provides a reasonable basis for our findings 
and conclusions based on our audit objectives. 



19 In 2010, the VA Office of Inspector General found issues with ORD's validation of some 
research expenditure data reported by investigators at the VA Maryland Health Care 
System facility. See Department of Veterans Affairs, Office of Inspector General, Health 
Care Inspection: Inappropriate Research and Development Data Entries Affecting 
Veterans Equitable Research Allocation (VERA) Funding VA Maryland Health Care 
System Baltimore, MD (Washington, D.C., Sept. 23, 2010). According to VA officials, the 
PTSD research funding data we obtained for this report were generated from a different 
data system than the data systems specifically examined by the VA Office of Inspector 
General. In addition, the PTSD research funding data are entered by VA officials at the 
central office and checked for accuracy by VA officials at both the central office and the 
facilities. 



Page 6 



GAO-11-32 Veterans Affairs Post-Traumatic Stress Disorder Research 



VA manages its intramural research program through ORD. According to 
ORD's 2009 to 2014 strategic plan, ORD has 10 research priority areas, 
which are topics of research that are considered important to VA. 20 The 
research priority areas are the health care needs of veterans who have 
served in Operation Enduring Freedom and Operation Iraqi Freedom, 
aging-related conditions, mental health care and well-being, chronic 
diseases, long-term care and caregiving, deployment-related exposure to 
hazardous environmental agents, equity in care, access in rural areas, 
women's health, and personalized medicine. According to VA officials, all 
of these research priority areas could include PTSD research. 

VA funds intramural research through the following: 

• VA's Merit Review Program: This program supports research studies 
typically conducted by one VA investigator at one VA facility and is 
administered by ORD's four research and development services, each of 
which has a different research focus. (See table 1.) Each research and 
development service is responsible for soliciting, reviewing, selecting, and 
funding research proposals submitted to the service. 



Table 1 : The Office of Research and Development's Research and Development Services 


Research and development 
service 


Focus of research 


Examples of research 


Biomedical Laboratory 


Biological or physiological principles in humans or 
animals 


Investigations of tissues, blood, or other 
biologic specimens 


Clinical Science 


Human beings 


Clinical studies examining interventions and 
effectiveness; epidemiological studies 


Health Services 


The interface of health care systems, patients, and 
health care outcomes 


Studies examining quality, access, patient 
outcomes, and costs 


Rehabilitation 


Improving the quality of life of impaired and 
disabled veterans 


Veteran rehabilitation studies; studies 
focused on identifying technical solutions for 
impaired and disabled veterans 



Source: GAO analysis of Department of Veterans Affairs information. 



20 In 2009, VA's Office of Inspector General found that appropriated funds for VA research 
addressed the broad spectrum of medical issues with which veterans contend. See 
Department of Veterans Affairs, Office of Inspector General, Healthcare Inspection: 
Review of the Veterans Health Administration's Use of Appropriated Funds for Research 
(Washington, D.C., 2009). 



Background 
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• VA's CSP: This program, which is administered by Clinical Science, funds 
larger-scale, multisite clinical trials and epidemiological research studies 
on key diseases that impact veterans. 



The Merit Review Program has research award funding limits, which are 
set by VA. In some cases, intramural research awards may only be funded 
for a certain number of years. See table 2 for more information. 



Table 2: Department of Veterans Affairs (VA) Intramural Research Study Funding Limits and Maximum Length of Award by 
Research and Development Service 


Service 


Funding limit per award per fiscal year 


Maximum length of award (in fiscal years) 




Pilot study 8 


Full study 


Pilot study 8 


Full study 


Biomedical Laboratory 


N/A 


$150,000" 


N/A 


4 


Clinical Science 


N/A 


$150,000 b 


N/A 


4 

(5 for clinical trials) 


Cooperative Studies 
Program 


N/A 


No limit 


N/A 


No limit 


Health Services 


$100,000 


$300,000° 


1 


4 


Rehabilitation 


$75,000 


$300,000 


2 


4 



Source: GAO analysis of VA information. 

°A pilot study is a study to establish feasibility or to develop data, a technique, a concept, or a 
procedure, which is preliminary to undertaking a full study. Two of VA's Office of Research and 
Development's research and development services — Health Services and Rehabilitation — accept 
pilot study proposals in addition to full study proposals. 

"According to VA officials, this limit may be exceeded for funding the salaries of investigators who are 
not VA clinicians or for equipment. 

According to VA officials, the total over all years may not exceed $925,000. 

In addition to individual studies conducted at VA facilities, VA has several 
research centers and programs that conduct or support PTSD research. 
For example, the National Center for PTSD focuses on PTSD research. VA 
also has Research Enhancement Award Programs, which help support 
PTSD research by providing staff and other resources to investigators. 
(For more information on VA research centers and research programs that 
conduct or support PTSD research, see app. I.) According to a VA official 
from the National Center for PTSD, VA does not fund most of the PTSD 
research that is being conducted today. 



Intramural research proposals may be service-directed — solicited by ORD 
on specific topics — or investigator-initiated — submitted by investigators 
to ORD on their own initiative. Investigators submit proposals either in 
response to a request for proposals on a specific topic (for service- 



Origin and Criteria for 
Intramural Research 
Proposals 
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directed proposals) or to an open request for proposals (for investigator- 
initiated proposals). For both the Merit Review Program and CSP, 
proposals are typically evaluated in two review cycles per year. 21 



To be considered for intramural research funding: 22 

• The proposal must be veteran-centric. 

• The proposal must have received approval from the director of the 
medical center and the research and development office of the medical 
center where the lead investigator, known as a principal investigator, is 
based. 23 

• The principal investigator and any coprincipal investigators must 
demonstrate a primary professional commitment to VA, as demonstrated 
by at least a 5/8 time VA appointment at the time the funding is awarded 
and previous VA experience, including experience in research and patient 



• Research must be conducted primarily on VA premises. The principal 
investigator and any coprincipal investigators must have designated 
research space within a VA medical center. 25 



21 According to VA, proposals may be reviewed outside of a review cycle depending on 
when ORD solicits requests for proposals. In rare cases, service-directed proposals are 
reviewed by ad hoc reviewers or solely by research and development service directors. 

22 Each research and development service may have additional requirements. 

23 The approval indicates that the medical center has agreed to commit the resources, such 
as space and staff, necessary to conduct the research study. 

24 Investigators who do not demonstrate a primary professional commitment to VA can 
request a waiver of this requirement. According to a VA document, a waiver may be granted 
by VA officials depending on the circumstances of the request, such as for a retired 
investigator who previously received funding through the Merit Review Program. 
According to VA, waivers are granted for about 35 investigators per year. 

25 The principal investigator and any coprincipal investigators may also have a designated 
research space in an approved non-VA facility or other VA facility, such as a community- 
based outpatient clinic. 
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PTSD Research 
Funding Increased 
from Fiscal Year 2005 
through Fiscal Year 
2009 



Overall intramural PTSD research funding from VA's medical and 
prosthetic research appropriation increased from $9.9 million in fiscal year 
2005 to $24.5 million in fiscal year 2009. The number of intramural PTSD 
research studies funded through the Merit Review Program and CSP 
increased from 47 in fiscal year 2005 to 96 in fiscal year 2009. 



VA Intramural PTSD 
Research Funding 
Increased from 
$9.9 Million in Fiscal Year 
2005 to $24.5 Million in 
Fiscal Year 2009 



Based on the VA data we obtained and summarized, we found that overall 
intramural PTSD research funding from VA's medical and prosthetic 
research appropriation increased from about $9.9 million in fiscal year 
2005 to about $24.5 million in fiscal year 2009, or by about 150 percent (see 
fig. 1). Overall intramural PTSD research funding included funding for 
specific PTSD studies as well as for other PTSD research-related funding, 
such as career development awards provided to junior VA investigators to 
conduct PTSD studies, salaries for VA investigators who are not VA 
clinicians, 26 funding for PTSD research conducted within ORD research 
centers, and PTSD research meetings. 



26 Salaries for VA investigators who are VA clinicians are paid from the medical services 
appropriation, one of the medical care appropriations. 
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Figure 1 : Department of Veterans Affairs (VA) Intramural Funding for Post- 
Traumatic Stress Disorder (PTSD) Research Studies and Other PTSD Research- 
Related Funding, Fiscal Years 2005 through 2009 
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Source: GAO analysis of VA data. 



Of the $80.2 million provided for PTSD studies from fiscal year 2005 
through fiscal year 2009, $51.3 million, or about 64 percent, was for studies 
funded through the Merit Review Program. The remaining approximately 
$28.9 million, or about 36 percent, was for CSP studies. (See fig. 2.) 



Page 11 



GAO-11-32 Veterans Affairs Post-Traumatic Stress Disorder Research 



Figure 2: Department of Veterans Affairs (VA) Intramural Post-Traumatic Stress 
Disorder (PTSD) Funding for Merit Review Program Studies and Cooperative 
Studies Program (CSP) Studies, Fiscal Years 2005 through 2009 
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Source: GAO analysis of VA data. 



From fiscal year 2005 through fiscal year 2009, intramural PTSD research 
funding ranged from 2.5 percent to 4.8 percent of VA's medical and 
prosthetic research appropriation. (See table 3 for VA intramural PTSD 
research funding and VA's medical and prosthetic research appropriations 
from fiscal year 2005 through fiscal year 2009.) For comparison, 27 
according to a 2009 report prepared by ORD staff for VA's National 
Research Advisory Council, 28 for fiscal year 2009, funding for intramural 
traumatic brain injury research was about $14.6 million, 2.9 percent of the 
medical and prosthetic research appropriation. Funding for spinal cord 



2 'Because of potential overlap in research areas, percentage estimates should not be added. 

2S The National Research Advisory Council is responsible for advising the Secretary of 
Veterans Affairs and the Under Secretary for Health on matters related to VA's research. 
The Office of Management and Budget requires that the National Research Advisory 
Council conduct an annual assessment of VA's intramural research program. 
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injury research was $27.2 million, 5.3 percent of the medical and 
prosthetic research appropriation. Funding for intramural cardiovascular 
disease and stroke research was $53.1 million, 10.4 percent of the medical 
and prosthetic research appropriation. 



Table 3: Department of Veterans Affairs (VA) Intramural Post-Traumatic Stress 
Disorder (PTSD) Research Funding and VA's Medical and Prosthetic Research 
Appropriation, Fiscal Years 2005 through 2009 


Dollars in millions 


Fiscal year 


VA intramural PTSD 
research funding 


VA's medical and 
prosthetic research 

appropriation Percentage 


2005 


$9.9 


$390.2 


2.5 


2006 


13.3 


412.0 


3.2 


2007 


15.7 


446.5 


3.5 


2008 


21.3 


480.0 


4.4 


2009 


24.5 


510.0 


4.8 


Total 


$84.7 


$2,238.7 


3.8 



Source: GAO analysis of VA data. 



Note: VA intramural research is also supported by VA's medical care appropriations, which are not 
represented in this table. For example, the medical care appropriations pay for the salaries of VA 
investigators who are VA clinicians (according to VA, about 70 percent of VA investigators are VA 
clinicians), administrative costs such as payroll and human resources, and logistical and 
infrastructure costs. 



The Number of PTSD 
Studies Funded through 
VA's Intramural Research 
Program Increased from 47 
in Fiscal Year 2005 to 96 in 
Fiscal Year 2009 



Similarly, we found that the number of PTSD studies funded from VA's 
medical and prosthetic research appropriations through VA's intramural 
research program increased from fiscal year 2005 through fiscal year 2009. 
(See fig. 3.) Specifically, in fiscal year 2005, 47 intramural PTSD research 
studies were funded while in fiscal year 2009, 96 intramural PTSD research 
studies were funded. 29 This represented an increase of more than 
100 percent. Of all the studies funded each fiscal year, only a small number 
were CSP studies. 



29 VA intramural research studies may be funded for multiple years (see table 2). Therefore, 
the same intramural PTSD research study funded in one fiscal year may also be funded the 
following fiscal year. In our analysis, this study would be counted as a funded study each 
fiscal year. In addition, some studies were not funded each fiscal year of their duration. For 
example, a study that was conducted from fiscal year 2006 through fiscal year 2009 may not 
have received funding in fiscal year 2007. 



Page 13 



GAO-11-32 Veterans Affairs Post-Traumatic Stress Disorder Research 



Figure 3: Number of Department of Veterans Affairs (VA) Intramural Post-Traumatic 
Stress Disorder (PTSD) Research Studies Funded per Fiscal Year, Fiscal Years 2005 
through 2009 
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Source: GAO analysis of VA data. 



VA Intramural PTSD 
Research Studies Are 
Funded Primarily 
According to 
Scientific Merit 



According to VA officials, intramural research proposals, including those 
on PTSD, are reviewed and funded in VA's Merit Review Program and VA's 
CSP primarily according to scientific merit. 



VA's Merit Review Program 

Intramural research proposals submitted to VA's Merit Review Program 
are reviewed through a series of steps prior to funding. See figure 4 for an 
overview of the submission, review, and funding process for proposals 
submitted to the Merit Review Program. (For more detailed information 
on this process, see app. II.) 
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Figure 4: Overview of Submission, Review, and Funding Process in Department of 
Veterans Affairs' (VA) Merit Review Program 
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Source: GAO analysis of VA documents and GAO interviews with VA officials. 



"Proposals are submitted electronically to grants.gov, the government's central grant identification 
and proposal portal. Proposals are then transferred to eRA Commons, an electronic system for grant 
administration functions, for VA processing and review. 

First, investigators submit proposals electronically. Investigators typically 
submit Merit Review Program proposals to grants.gov, the government's 
central grant identification and proposal portal, 30 in response to a request 
for proposals. Submitted proposals are then transferred to eRA Commons, 
an electronic system for grant administration functions, for VA processing 
and review. 31 

Second, each proposal is assigned to a merit review panel for evaluation. 
Each merit review panel reviews proposals in a specific research topic 
area, and is composed of panelists, typically associate-level professors, 
who are selected based on their expertise in this area. 32 According to VA 
documents, as of 2010, there were a total of 35 merit review panels across 
VA's research and development services. 33 The merit review panels 
evaluate each proposal based on its scientific merit. 34 Panelists consider 



"Depending on the specific requirements of the request for proposal, a letter of intent (a 
preliminary outline of a research proposal) may be required prior to the submission of a 
research proposal. 

'"Electronic research administration, or eRA Commons, is managed and supported by the 
National Institutes of Health to provide support for the funding of research proposals. 

"Merit review panelists may or may not be employed by VA. According to VA officials, 
most merit review panelists have previous experience reviewing proposals. According to a 
VA document, the number of panelists on a merit review panel may vary. For example, the 
number of panelists on Rehabilitation's merit review panels ranges from 16 to 59. 

'"Biomedical Laboratory and Clinical Science share the same merit review panels. 

34 In addition, for Health Services, along with scientific merit, reviewers take into account 
research priority areas of Health Services when scoring proposals. 
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several criteria in evaluating the overall scientific merit of a proposal. (See 
table 4 for criteria used to determine scientific merit.) 



Table 4: Scientific Merit Scoring Criteria in the Department of Veterans Affairs' (VA) 
Merit Review Program 



Criteria Description 

Significance Supports/advances the health and health care of veterans and research 
field in general; addresses important scientific question/area; makes a 
potential contribution to scientific literature. 

Approach Incorporates current scientific/theoretical bases; is hypothesis-driven; 

uses appropriate research design/methods for addressing hypothesis; 
ensures that feasibility of methods is clear. 

Innovation Addresses new concepts, gaps, or both, in the research area; addresses 
potential for impact of findings on existing field of research, treatment 
paradigms, or both. 

Environment Has appropriate knowledge/background and resources (e.g., equipment 
and staff) to ensure completion of study. 

Feasibility Provides sufficient evidence to determine that the proposed study can be 
successfully conducted and completed. 

Investigator Ensures that investigators involved in the proposed study are 

appropriately trained and have expertise in the proposed area of 
research. 



Source: GAO analysis of VA information. 

Third, the merit review panelists score the proposals to determine their 
rank. Each panelist provides a score to each of the proposals reviewed by 
the panel. The scores are averaged to create a "priority score" for the 
proposal. 35 (See app. II for specific scoring guidelines given to panelists in 
all research and development services.) All proposals scored by the merit 
review panel are then ranked by priority score among all of the proposal 
scores recently assigned by the merit review panel. The rank of the 
proposal is used to determine the "percentile" of the proposal. 36 



36 For Health Services, prior to assigning a priority score for a proposal, merit review 
panelists vote on whether to approve, conditionally approve, or reject the proposal. 
Conditional approval means the proposal is approved pending additions or changes. Only 
approved and conditionally approved proposals are assigned a priority score. 

36 According to VA, the percentile is calculated by the proposal's rank, determined by its 
priority score in comparison to the scores of the current and previous three review cycles, 
minus 0.5, divided by the total number of applications being considered. For example, if a 
proposal received the fifth best priority score and there were a total of 60 proposals 
reviewed over the four cycles, the percentile for that application would be 5 minus 
0.5 divided by 60, equaling .075. 
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Finally, research and development service directors determine how many 
proposals to fund. All of the proposals scored by all merit review panels in 
a research and development service in the review cycle are ranked 
together by their percentiles to be considered for funding. According to VA 
officials, research and development service directors typically fund up to 
the 25th percentile of proposals in a review cycle, beginning with those 
with the most scientific merit, although the number of proposals funded 
may vary depending on the budget. According to VA, research and 
development service directors may also choose to fund a small number of 
additional proposals at the margin that respond to research priority areas. 
For example, if the fundable range determined by a research and 
development service director was up to the 25th percentile, proposals at 
the 26th percentile related to research priority areas could also be 
considered for funding. 37 

VA's Cooperative Studies Program 

VA intramural research proposals submitted to CSP are reviewed and 
scored in a process similar to that of the Merit Review Program prior to 
consideration for funding. To help develop the CSP proposal, investigators 
are assisted by members of a CSP center, 38 a VA entity that provides 
guidance and support for research across multiple sites. (See fig. 5 for an 
overview of the process for submitting, reviewing, and funding a CSP 
proposal. For more information on ORD's CSP review process, see 
app. III.) 



37 After a proposal is approved for funding by the research and development service 
director, the proposal must undergo review by an institutional review board, an 
independent committee that reviews research based on ethical considerations. In addition, 
depending on the topic of the proposal, it may be reviewed by additional committees at this 
time, such as a data monitoring committee. 

;i8 There are nine CSP centers: VA operates five CSP Coordinating Centers to support 
multisite clinical trials and four Epidemiological Research and Information Centers to 
support epidemiology studies. In addition to receiving support from a designated CSP 
center, CSP proposals that involve drugs or medical devices also receive assistance from 
the CSP Clinical Research Pharmacy Coordinating Center, and CSP proposals that involve 
economic analysis receive support from VA's Health Economics Resource Center. 
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Figure 5: Overview of Submission, Review, and Funding for a Department of 
Veterans Affairs' (VA) Cooperative Studies Program (CSP) Proposal 
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Source: GAO analysis of VA documents and GAO interviews with VA officials. 



"Prior to submitting the proposal to CSP central office, the principal investigator must have a letter of 
intent approved by the Director of Clinical Science. 

Before submitting a research proposal, investigators submit a letter of 
intent, or a preliminary outline of a proposal, to the Director of Clinical 
Science to be approved for planning a CSP proposal. 39 Based on the merit 
of the letter of intent, as determined by three or more reviewers, 40 the 
Clinical Science Director decides whether to fund planning efforts to 
develop a CSP proposal. 41 

When the principal investigator receives approval to begin planning 
efforts, the Clinical Science Director assigns a CSP center to provide 
statistical and methodological guidance to the investigator. The director of 
the CSP center designates a project manager and methodologist, such as a 
person with expertise in biostatistics, to provide guidance to the principal 
investigator. The Clinical Science Director, with recommendations from 
the principal investigator, then forms a planning committee of additional 
experts to assist in developing a CSP proposal. The planning committee 
develops a CSP proposal over the course of two planning meetings. 

Once a proposal is developed, the CSP center, on behalf of the principal 
investigator, submits a hard copy proposal to the CSP central office for 



J9 A letter of intent typically contains an outline of the proposed research study, including 
justification for multiple sites where the research will be conducted and a list of experts 
who can assist in the initial planning efforts of the research study. 

"Reviewers of letters of intent are selected based on their expertise in areas related to the 
topic of the letter of intent. 

"Planning efforts include travel funds for personnel involved in developing the proposal. 
According to VA, there are two planning meetings that involve approximately 15 people. 
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evaluation by the Cooperative Studies Scientific Merit Review Board. 42 
This board consists of reviewers who have extensive experience in clinical 
research and the conduct of clinical trials or epidemiology studies. 43 
Reviewers evaluate CSP proposals based on scientific merit. According to 
VA, the scientific merit of a CSP proposal is defined by the importance of 
the proposal, its feasibility, the clarity and achievability of its objectives, 
the adequacy of the plan of investigation, the correctness of the technical 
details, and the adequacy of the safeguards for the welfare of the 
patients. 44 Based on these criteria, reviewers discuss the general scientific 
merit of the proposal. Reviewers vote on whether to unconditionally 
approve, conditionally approve, reject or defer with recommendation for 
resubmittal, or reject each proposal. (See table 5 for an overview of 
funding recommendations provided by the board.) 



Table 5: Overview of Funding Recommendations Provided by the Department of 
Veterans Affairs' (VA) Cooperative Studies Scientific Merit Review Board 



Recommendation for funding 

Unconditional approval 



Conditional approval 



Rejection or defer with recommendation 
for resubmittal 



Rejection 



Source: GAO analysis of a VA document. 



Definition 

The proposal is approved without changes or 
additions to the proposal. 

The proposal is approved pending certain 
changes or additions to the proposal. 

The proposal is considered to be worthwhile, 
but is in need of major revisions. A revised 
proposal can be resubmitted to the Clinical 
Science Director. 

The proposal is rejected. The principal 
investigator must submit a new request for 
planning to submit a revised proposal. 



42 CSP proposals are currently not submitted electronically; however, ORD officials estimate 
that electronic submission of CSP proposals through eRA Commons will begin in fall 2011. 

4 According to a VA document, members of the Cooperative Studies Scientific Merit Review 
Board serve 4-year terms, and ad hoc members can be added depending on specific 
expertise that may be needed to review a proposal. As of October 2010, there were six 
members of the Cooperative Studies Scientific Merit Review Board. 

44 CSP proposals are also reviewed by a human rights committee at the designated CSP 
center to determine if the protection of patients' rights and welfare is adequate. CSP 
proposals must be approved by a human rights committee prior to being funded. 
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After the reviewers vote, they each provide scores for a proposal 
recommended for funding based on scientific merit. 45 The scores are then 
averaged to provide a priority score for a proposal. Finally, the Clinical 
Science Director considers the priority scores of all the proposals in that 
review cycle and selects the proposals with the strongest priority scores 
for funding. According to VA officials, the number of proposals funded 
may vary depending on the budget. 46 



VA Has Established a 
Standardized Process 
to Review All 
Relevant PTSD 
Research Outcomes 
and Incorporate Them 
into a PTSD CPG 



The VA/DOD Evidence-Based Practice Work Group, which is responsible 
for developing and updating all of VA's CPGs, has a standardized and 
reproducible process to review all relevant research outcomes when 
developing or updating all CPGs, including the PTSD CPG. To develop or 
update a CPG, the VA/DOD Evidence-Based Practice Work Group 
identifies and assigns a group of VA and DOD clinical leaders and experts 
who are knowledgeable in the subject area to work on the CPG. 47 
Generally, the process to develop or update a CPG consists of the 
following steps. 

First, the assigned group of VA and DOD clinical leaders and experts 
identifies "clinical questions" that will be answered in the CPG. According 
to VA officials, clinical questions can be either broad or specific. For 
example, the 2004 PTSD CPG contained clinical questions regarding 
whether early intervention is more effective than later intervention, and 
whether certain interventions, such as different psychotherapies, are more 
effective than others. 



• Second, in order to minimize bias, an external contractor conducts a 
systematic review of relevant research and selects and summarizes the 
most methodologically rigorous research studies that are applicable to 
each of the clinical questions. 



45 Scores are given only to conditionally and unconditionally approved proposals. 

46 After a CSP proposal has received approval for funding, the proposal must undergo 
additional steps before study subjects are enrolled. For example, the proposal must be 
reviewed by an institutional review board, and an executive committee meets to review the 
operational and monitoring aspects of the study. 

47 In addition, representatives from VA's Office of Patient Care Services are responsible for 
identifying VA and DOD clinical leaders. 
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• Third, after receiving summaries of the studies with the highest level of 
evidence, the VA and DOD group of clinical leaders and experts rates the 
research using an established grading scheme that considers 

• the level of evidence of each research study — the scope and 
methodological rigor of an individual study; 

• the overall quality of evidence — the overall quality of all of the 
research that addresses a particular clinical question, considering the 
level of evidence of all the studies considered; and 

• the net effect of an intervention — according to the collective results of 
the studies considered, the intervention's benefits minus the 
intervention's harms. 

• Finally, the assigned group of VA and DOD clinical leaders and experts 
assigns a grade to each evidence-based recommendation based on an 
assessment of the overall quality of evidence and the net effect of the 
intervention. (See app. IV for a detailed description of the process used to 
develop evidence-based VA/DOD CPGs.) 

The process for conducting a systematic review of research outcomes to 
develop or update a CPG is repeated as often as is deemed necessary by 
the VA/DOD Evidence-Based Practice Work Group according to its written 
procedures and designated time frames. 48 According to VA/DOD Evidence- 
Based Practice Work Group documents, routine updates to the CPGs 
should ideally occur approximately every 2 years. However, updates to 
CPGs often do not occur every 2 years, and VA officials told us that some 
CPGs are updated more frequently than others based on availability of 
resources and priority areas. Additionally, VA officials reported that a CPG 
will be immediately updated if any evidence-based recommendation 
contained in it is identified as harmful to patients. 

According to VA, the VA/DOD Evidence-Based Practice Work Group 
approved an update to the 2004 PTSD CPG on October 25, 2010, and 
published the update on VA's Web site on November 17, 2010. 49 According 



48 According to VA, the development or update of a CPG typically takes 18 months. 

49 According to VA, the update of the PTSD CPG mainly strengthens the evidence that was 
already contained in the 2004 PTSD CPG. According to a VA official, the updated PTSD 
CPG will include a module on complementary alternative medicine, which was not 
included in the 2004 PTSD CPG. 
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to VA officials, the systematic process outlined above was used to review 
all relevant research outcomes and make evidence-based 
recommendations for PTSD services to both develop and update the PTSD 
CPG. 



VA Reviewed 
Research Outcomes 
and Used Existing 
Resources in 
Determining Which 
PTSD Services to 
Require Its Facilities 
to Provide or Make 
Available to Veterans 



According to VA officials, the decision to require that cognitive processing 
therapy and prolonged exposure therapy be made available to veterans 
diagnosed with PTSD at VA facilities — as indicated in the Handbook, 
which established certain requirements for mental health services within 
VA — was based on a review of research outcomes and the availability of 
existing resources. 

Review of research outcomes. According to VA, agency officials and 
qualified subject matter experts reviewed relevant research outcomes and 
the quality of the research to determine the most efficacious PTSD 
treatments available when determining which PTSD services to include in 
the Handbook and make available to veterans. Specifically, VA officials 
told us that their decision to include cognitive processing therapy and 
prolonged exposure therapy in the Handbook was influenced by the fact 
that both of these had been graded as level "A" treatments in the 2004 
PTSD CPG (indicating that the intervention is always indicated and 
acceptable). 50 Furthermore, VA officials said that these two therapies had 
greater evidence supporting their effectiveness than other PTSD services 
also graded as level "A" in the 2004 PTSD CPG. 51 In addition, VA officials 
added that their decision was validated by the results of a VA- 
commissioned Institute of Medicine study published in 2008 that reviewed 
the evidence for existing PTSD treatments. According to VA, the study 
found that cognitive processing therapy and prolonged exposure therapy 
were considered efficacious treatments for PTSD. 52 While the Institute of 



""in the CPG, cognitive processing therapy is graded a level "A" treatment specifically for 
female sexual assault-related PTSD. However, VA officials told us that they also consider 
cognitive processing therapy to be effective for PTSD in veterans. See app. IV for a detailed 
description of the process used to determine grades for evidence-based recommendations. 

Dl Two other psychotherapies, stress inoculation training and eye movement desensitization 
and reprocessing, were also graded as level "A" treatments for PTSD in the 2004 PTSD 
CPG, but were not included in the Handbook. 

° 2 Institute of Medicine, Treatment of Posttraumatic Stress Disorder: An Assessment of the 
Evidence (Washington, D.C., 2008). The Institute of Medicine is an independent, nonprofit 
organization that works outside of government to provide authoritative advice to decision 
makers and the public. The Institute of Medicine found that exposure therapies, including 
prolonged exposure therapy and elements of cognitive processing therapy, are efficacious 
treatments for PTSD. 
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Medicine report was released after VA had already decided to include 
cognitive processing therapy and prolonged exposure therapy in the 
Handbook, VA officials explained that the Institute of Medicine report was 
the basis for the decision not to include other PTSD services in the 
Handbook. 

• Availability of existing resources. VA officials told us that prior to 
issuing the Handbook in 2008, VA had already begun investing 
considerable resources to implement national training programs for 
cognitive processing therapy and prolonged exposure therapy in 2006 and 
2007, respectively. VA officials said that they decided to implement the 
national training programs because VA realized the need to create 
sufficient capacity so that evidence-based PTSD treatments could be 
available to veterans throughout the VA system. VA explained that the 
national training programs were rolled out in advance of the Handbook's 
issuance as part of the implementation of VA's Comprehensive Veterans 
Health Administration Strategic Plan for Mental Health Services, which 
called for rapid implementation of evidence-based treatments. VA did this 
to ensure that it had the capacity to provide cognitive processing therapy 
and prolonged exposure therapy to all veterans with PTSD for whom these 
treatments were clinically appropriate. 53 VA officials said that they were 
able to begin implementing national training programs for cognitive 
processing therapy in 2006 and prolonged exposure therapy in 2007 
because VA had qualified instructors to administer the programs and 
money available to fund them. 54 



D<I VA officials reported that incorporating cognitive processing therapy and prolonged 
exposure therapy into the Handbook helped ensure that clinicians who had been trained in 
these therapies prior to development of the Handbook would be able to offer and provide 
them in VA facilities as needed. In May 2010, VA's Office of Inspector General reported that 
as of November 17, 2009, the total number of mental health practitioners who had 
completed either the cognitive processing therapy or the prolonged exposure therapy 
training was 3,086. See Department of Veterans Affairs, Office of Inspector General, 
Healthcare Inspection: Progress in Implementing the Veterans Health Administration's 
Uniform Mental Health Services Handbook (Washington, D.C., 2010). 

d4 To help implement VA's Comprehensive Veterans Health Administration Strategic Plan 
for Mental Health Services (November 2004), VA allocated additional resources to fund 
mental health strategic plan initiatives in fiscal years 2005 and 2006. According to VA 
officials, a part of these additional resources was used to pay for the national cognitive 
processing therapy and prolonged exposure therapy training programs until the costs of 
the programs could be integrated into VA's congressional budget submission. 
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Unlike the written and standardized process that the VA/DOD Evidence- 
Based Practice Work Group established to develop CPGs, VA does not 
have a formal written process or framework to explain its decision for 
including cognitive processing therapy and prolonged exposure therapy in 
the Handbook. 55 VA officials explained that they followed a process when 
choosing cognitive processing therapy and prolonged exposure therapy, 
but added that clinical decision-making processes are not typically 
expected to be documented in a formal manner. 

VA officials told us that they plan to assess the implementation of the 
Handbook and will update PTSD requirements in it as needed or as new 
information or unexpected obstacles arise in the future. 56 VA officials 
stated that they are currently clarifying the language regarding some of the 
requirements, but do not plan to revise any of the requirements relating to 
PTSD services at this time. 



ASGFICV Comments ^ e P rov *ded a draft of this report to VA and received technical comments, 

° which we incorporated into our report as appropriate. 

We are sending a copy of this report to the Secretary of Veterans Affairs. 
The report also is available at no charge on the GAO Web site at 
http://www.gao.gov. 

If you or your staff have any questions about this report, please contact me 
at (202) 512-7114 or williamsonr@gao.gov. Contact points for our Offices 



o5 However, VA officials provided documentation regarding consultation with experts and a 
list of VA entities that participated in reviewing the Handbook. 

* According to VA officials, VA policies must be recertified every 5 years to remain in effect. 
The Handbook must be recertified on or before September 2013. 
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of Congressional Relations and Public Affairs may be found on the last 
page of this report. GAO staff members who made key contributions to 
this report are listed in appendix V. 

Sincerely yours, 




Randall B. Williamson 
Director, Health Care 



Enclosures - V 
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Appendix I: VA Research Centers and 
Programs That Conduct or Support PTSD 
Research 



In addition to post-traumatic stress disorder (PTSD) research studies that 
are conducted by individual Department of Veterans Affairs (VA) 
investigators, or researchers, VA also funds a number of research centers 
or programs that conduct or support PTSD research. See table 6 for a 
description of these VA research centers and programs. 



Table 6: Department of Veterans Affairs (VA) Research Centers and Programs That Conduct or Support Post-Traumatic 
Stress Disorder (PTSD) Research 


VA research centers/programs 


Description 


Established 
by Congress? 


Number of centers 
or programs 
with a primary 
focus on PTSD 


Office of Mental Health Services 


The National Center for PTSD 


Advances the clinical care and social welfare 
of veterans through research, education, and 
training in the science, diagnosis, and 
treatment of PTSD and stress-related 
disorders. 


Yes 


r 


Centers of Excellence 


Research the causes and treatments of 
mental disorders and use the dissemination 
of education to implement new knowledge 
into routine VA clinical practices. 


Yes 


2 


Mental Illness Research, Education, and 
Clinical Centers 


Research the causes and treatments of 
mental disorders and use education to 
implement knowledge into routine clinical 
practice in VA. 


Yes 


4 


Office of Research and Development 


Centers of Excellence 


Create or maintain a core program of 
investigators to support and facilitate the 
development of research studies and train 
and mentor new investigators. 


No 


8 


Quality Enhancement Research Initiative 
Program 


Enhances the quality and outcomes of VA 
health care by implementing research 
findings and evidence-based 
recommendations into routine clinical 
practice. 


No 


1 b 


Research Enhancement Award Programs 


Support groups of VA investigators that 
address specific medical problems of 
veterans to promote innovative research, 
train new medical research investigators, 
and foster new research collaborations 
among investigators. 


No 


1 


Source: GAO analysis of VA information. 



"The National Center for PTSD, headquartered in White River Junction, Vermont, has seven divisions 
in five locations across the country. 
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"Although the Mental Health Quality Enhancement Research Initiative (one of nine Quality 
Enhancement Research Initiative Centers) focuses on depression and schizophrenia, Office of 
Research and Development officials stated that within the past few years it has become more 
involved with PTSD research. 
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Research proposals submitted to the Department of Veterans Affairs' (VA) 
Merit Review Program are evaluated in merit review panels that each 
review proposals in a specific research topic area. Each merit review 
panel is comprised of panelists, typically associate-level professors, who 
are selected based on their expertise in the area. Panelists are responsible 
for scoring proposals based on scientific merit to provide funding 
recommendations. See figure 6 for a detailed description of the Merit 
Review Program's process for reviewing research proposals and table 7 for 
the Merit Review Program's scoring guidelines. 
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Figure 6: Summary of the Department of Veterans Affairs' (VA) Merit Review 
Process for the Merit Review Program 



1 . The portfolio manager, an official responsible for managing a specific area of research, 
assigns three panelists as the primary, secondary, and tertiary panelists for each research 
proposal. These panelists are responsible for reviewing the proposal prior to the panel meeting, 
and individually submitting a preliminary score and summary of the proposal into eRA 
Commons, an electronic system for grant administration functions, for VA processing and 
review. The preliminary scores range from 1.0 and 5.0 (with scores closer to 1.0 being more 
meritorious). 



2. The primary, secondary, and tertiary panelists present their preliminary scores to the entire 
panel 3 and respond to feedback provided by the other panelists about the proposal. 11 



3. Based on the discussion at the panel meeting, the primary, secondary, and tertiary panelists 
can change their preliminary scores. The primary, secondary, and tertiary panelists must 
announce to the panel the final score each plans to give the proposal. The final scores also 
range from 1 .0 and 5.0 (with scores closer to 1 .0 being more meritorious). 



4. The portfolio manager asks if any panelist plans to score the research proposal 0.3 points 
above or below the range of the final scores provided by the primary, secondary, and tertiary 
panelists. If any panelist intends to score outside of this range, the panelist must discuss this 
decision with the rest of the panel and explain reasons for doing so. 



5. All panelists on the review panel then submit a final score for the research proposal. The 
average of the final scores multiplied by 1 00 is known as the priority score. In addition, the 
primary panelist writes a summary of the discussion of the panel meeting and submits the 
summary into eRA Commons. 



Source: GAO analysis of VA documents and GAO interviews with VA officials. 

"According to VA's policy, panelists who have a conflict of interest with a given proposal are required 
to leave the meeting before the proposal is discussed and do not score the proposal. According to 
VA, a conflict of interest exists when a reviewer has an interest in a research proposal that is likely to 
bias his or her evaluation of it. Panelists are expected to inform the portfolio manager if they have a 
conflict of interest. 

"In some instances, research proposals are streamlined, meaning that they are evaluated only by the 
primary, secondary, and tertiary panelists, and are not discussed by the entire merit review panel. 
Proposals may be streamlined if the average of the primary, secondary, and tertiary panelists' 
preliminary scores falls in the upper half of the range of scores, indicating less merit. According to VA 
officials, proposals with a preliminary score of lower than 2.2, indicating very good or excellent, are 
also streamlined for two research and development services. (See table 7 for merit review process 
scoring guidelines.) In these instances, the administrator of the merit review panel meeting asks the 
panelists if anyone has an objection to streamlining the proposal. If a panelist objects, the proposal is 
discussed in the merit review panel meeting and it is not streamlined. Streamlined proposals with 
preliminary scores that fall in the upper half of the range of scores, indicating less merit, are not 
considered for funding. 
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Table 7: Scoring Guidelines for the Department of Veterans Affairs' (VA) Merit Review Program 



Scoring range and grade Description 

1 .0 - 1 .5 Proposed research addresses important scientific area that lacks needed knowledge. Hypothesis(es) 

Excellent is clearly stated and research design/methodology is appropriate. Research is innovative, 

representing state-of-the-art science. Potential findings may have a vital role in advancing the health 
and health care of veterans and research in general. Resources listed suggest a very high probability 
of the study's completion. 

Proposed research addresses important scientific area. Hypothesis(es) is clearly stated and the 
research design/methodology is appropriate with a few minor exceptions. Potential findings may 
have an important role to the health and health care of veterans and research in general. Resources 
listed suggest a high probability of the study's completion. 



Proposed research addresses a valid area of investigation. Hypothesis(es) is clearly stated, but 
research design/methodology contains key flaws that should be corrected. Potential findings may 
contribute to the health and health care of veterans and research in general. Resources listed 
suggest that the study could be completed. 

Proposed research requires further preliminary data to warrant investigation as a viable area of 
research. Hypothesis(es) is not clear, research design/methods contain significant flaws, or both. It is 
not clear how potential findings would contribute to the health and health care of veterans and 
research in general. It is unclear whether the resources listed are sufficient to ensure study 
completion. 

3.5 - 5.0 Proposed research does not appear to address an important scientific question/area. Hypothesis(es) 

Poor is not clearly stated, research design/methodology is inappropriate or contains uncorrectable flaws, 

or both. Design/methodological limitations hinder any significant conclusions that would contribute to 
the health and health care of veterans, research in general, or both. Resources listed do not suggest 
that the study will be completed. 



Source: GAO analysis of VA documents. 
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1.6-2.2 

Very good 



2.3-2.8 
Good 



2.9-3.4 
Fair 



Appendix III: The Review Process for VA's 
Cooperative Studies Program 



Research proposals submitted to the Department of Veterans Affairs' (VA) 
Cooperative Studies Program (CSP) are reviewed and scored by the 
Cooperative Studies Scientific Merit Review Board. Reviewers on the 
board are chosen based on their expertise in clinical or epidemiological 
research. They typically serve 4-year terms, and ad hoc members can be 
added depending on specific expertise that may be needed to review a 
proposal. According to VA, as of October 2010, there were six reviewers 
on the board. 

During the research proposal review process, the study team — which 
includes the lead researcher (referred to as the principal investigator) and 
a methodologist, such as a person with expertise in biostatistics — has an 
interactive discussion with the board regarding the proposal. 1 Reviewers 
evaluate CSP proposals based on scientific merit and provide scores to 
reflect their funding recommendations. See figure 7 for a detailed 
description of the review process for CSP research proposals. 



'The study team may also include a health economist, in cases where a CSP proposal 
involves economic analysis. In addition, the principal investigator may ask the Clinical 
Science Director to allow other consultants to be a part of the study team, as necessary. 
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Figure 7: Summary of the Department of Veterans Affairs' (VA) Cooperative Studies 
Program (CSP) Review Process 



1. The Deputy Director of CSP assigns three individuals on the Cooperative Studies 
Scientific Merit Review Board to be the initial reviewers for a CSP proposal. One of the initial 
reviewers is designated as the primary reviewer. All three initial reviewers prepare written 
reviews of the proposal based on scientific merit prior to the board meeting. 3 



2. A first board meeting is held to summarize and discuss the key critiques of the written 
reviews provided by the initial reviewers. The primary reviewer is responsible for introducing 
the proposal to the board. The study team, which includes the principal investigator and a 
methodologist, such as a person with expertise in biostatistics, is not present at this meeting. 



3. A second board meeting is held with the study team. b At the beginning of this meeting, the 
principal investigator is provided a summary of the critiques discussed in the first board 
meeting. The study team is then given 15 minutes to provide a summary of the proposal, 
including its importance to VA, and to address the critiques. 



1 


r 


4. After the study team responds to the board, the board and the study team engage in an 
interactive discussion regarding the proposal. 




r 



5. After the interactive discussion, the study team is dismissed and the board votes on 
whether to unconditionally approve, conditionally approve, reject or defer with 
recommendation for resubmittal, or reject funding of the proposal. d For proposals that are 
conditionally and unconditionally approved, each member of the board assigns a numeric 
rating from 10 to 50, with lower scores indicating more meritorious proposals, based on 
scientific merit. The average of the scores for each proposal is considered the priority score. 9 



Source: GAO analysis of VA documents and GAO interviews with VA officials. 

"According to a VA document, the primary reviewers are typically a biostatistician and a clinician. 
Reviewers are asked to comment on the importance of the proposal, its feasibility, the clarity and 
achievability of its objectives, the adequacy of the plan of investigation, the correctness of the 
technical details, the adequacy of the safeguards for the welfare of the patients, and any other 
pertinent features of the proposal. According to VA, these criteria are components of scientific merit. 

"For CSP proposals that involve economic analysis, a health economist is also present at the board 
meeting to discuss the proposal. In addition, the principal investigator may ask the Clinical Science 
Director to allow consultants to be present at the board meeting if review of the proposal requires 
expertise in a specific area. 

"According to VA, if there are any economic issues raised relating to the proposal, the study team will 
be given an additional 5 minutes to address these issues with a health economist. 
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"According to a VA document, unconditional approval means a proposal is approved by the board 
without changes and is recommended for funding. Conditional approval means that the proposal is 
approved by the board pending certain changes or additions to the proposal. Reject or defer with 
recommendation for resubmittal means the board considers the proposal to be worthwhile, but it is 
need of major revisions. In this case, if the principal investigator chooses to submit a revised 
proposal, the Clinical Science Director may waive the requirement to submit a letter of intent. If the 
board rejects the proposal, the principal investigator must submit a new letter of intent to resubmit a 
revised proposal. 

"After the board scores the proposal, a human rights committee typically provides a general 
assessment of the protection of patients' rights and welfare as described in the proposal. For studies 
with unique ethical considerations, this review may occur prior to the review by the Cooperative 
Studies Scientific Merit Review Board. 
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In 1999, the Department of Veterans Affairs (VA) and the Department of 
Defense (DOD) formed the VA/DOD Evidence-Based Practice Work Group 
to issue joint VA/DOD clinical practice guidelines (CPG) — tools that 
provide guidance and evidence-based recommendations to clinicians 
regarding the most effective interventions and services for a variety of 
health care topics. 1 To develop or update a CPG, the VA/DOD Evidence- 
Based Practice Work Group has a standardized process to ensure that 
systematic reviews of relevant research outcomes are conducted in order 
to formulate evidence-based recommendations for prevention, 
assessment, and treatment services. 

To develop or update a CPG, the VA/DOD Evidence-Based Practice Work 
Group identifies two clinical leaders — one from VA and one from DOD — 
who then help identify not more than 15 to 20 other experts in the subject 
area to form a "guideline working group." 2 A member of the VA/DOD 
Evidence-Based Practice Work Group is also selected to be an evidence 
chaperone for each CPG to ensure that conformity to prevailing standards 
for conducting high-quality systematic reviews is upheld. 3 

To determine the scope of the CPG, the guideline working group, the 
evidence chaperone, and a facilitator are responsible for identifying 
clinical questions that are to be answered by a systematic review of 
relevant research outcomes. 4 According to VA officials, clinical questions 



'Evidence-based care refers to approaches that have consistently been shown in controlled 
research to be effective for a particular condition or conditions. The VA/DOD Evidence- 
Based Practice Work Group makes decisions about which clinical practice guidelines for 
specific conditions will be developed and oversees their development. As of August 2010, 
the VA/DOD Evidence-Based Practice Work Group had issued 24 CPGs on conditions such 
as chronic heart failure and major depressive disorder. Since VA and DOD issue the CPGs 
in a joint effort, they are intended to be used by both VA and DOD clinicians. The VA/DOD 
Evidence-Based Practice Work Group is supported by VA's Office of Quality and 
Performance and the U.S. Army Medical Department's Office of Quality Management. 

Tn addition, representatives from VA's Office of Patient Care Services are responsible for 
identifying VA and DOD clinical leaders. 

! VA and DOD designees from the VA/DOD Evidence-Based Practice Work Group, with 
assistance from VA's Office of Patient Care Services, are responsible for selecting the VA 
and DOD clinical leaders and the evidence chaperone. In addition, they also select another 
member of the VA/DOD Evidence-Based Practice Work Group to monitor the development 
of the CPG. 

4 The facilitator is also responsible for keeping the guideline working group focused during 
meetings and to ensure that it focuses on the quality of research outcomes when 
formulating evidence-based recommendations in the CPG. 
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can be both broad and specific. For example, the 2004 post-traumatic 
stress disorder CPG contained clinical questions regarding whether early 
intervention is more effective than later intervention and whether certain 
interventions, such as different psychotherapies, are more effective than 
others. 5 

According to VA, in order to answer these clinical questions, an external 
evidence center — an entity that conducts systematic reviews of research 
on a variety of topics — is contracted to collect and review all relevant 
research (including, but not limited to, VA- and DOD-sponsored research) 
to assess its applicability to each clinical question under consideration 
using explicit and reproducible methods. 6 The evidence center then 
focuses its review on the best available research, that is, high-quality, 
methodologically rigorous studies that address health issues that impact 
VA and DOD populations and consider the effectiveness as well as the 
harms and benefits of the intervention at issue. According to VA officials, 
the evidence center provides summaries of only the best available 
research to the guideline working group for review. 7 

After receiving the summaries, the guideline working group reviews the 
research in sequential steps using an established rating scheme developed 
by the U.S. Preventive Services Task Force to formulate evidence-based 
recommendations. 8 See figure 8 for an overview of the steps that the 
guideline working group uses to formulate evidence-based 
recommendations. 



"The post-traumatic stress disorder CPG is formally known as The VA/DOD Clinical 
Practice Guideline for the Management of Post-Traumatic Stress (January 2004). 

6 According to a VA official, an external evidence center is used to minimize bias. 

7 The evidence center provides summaries of research of lower quality only if no higher- 
quality research is available. 

s The U.S. Preventive Services Task Force was first convened by the Department of Health 
and Human Services' Public Health Service in 1984, and since 1998 has been sponsored by 
the Department of Health and Human Services' Agency for Healthcare Research and 
Quality. The U.S. Preventive Services Task Force uses explicit criteria to grade the 
scientific evidence for a broad range of clinical services and develop evidence-based 
recommendations for clinicians and health systems. While the U.S. Preventive Services 
Task Force updated its criteria in May 2007, the VA/DOD Evidence-Based Practice Work 
Group currently uses U.S. Preventive Services Task Force criteria developed prior to May 
2007. 
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Figure 8: Overview of the Process Used by the Guideline Working Group 
Established by the Department of Veterans Affairs (VA)/Department of Defense 
(DOD) Evidence-Based Practice Work Group to Grade Evidence-Based 
Recommendations for Clinical Practice Guidelines 



Relevant research is 
assessed by its level of 
evidence 



Overall quality of 
evidence that addresses 
a particular clinical 
question is assessed 



The net effect of the 
intervention is assessed 
(benefits minus harms) 



Source: GAO analysis of VA/DOD documents and GAO interviews with VA officials. 



J 



> 



Evidence-based 
recommendations are 
graded (based on overall 
quality of evidence + net 
effect of the intervention) 



Level of evidence. First, the guideline working group reviews the 
summaries to identify the level of evidence, or the level of methodological 
rigor. For example, research studies that have the highest quality are 
categorized as "I" (indicating at least one properly done randomized 
controlled trial), while research studies of the lowest quality are 
categorized as "III" (indicating that the research reflects the opinion of 
respected authorities, descriptive studies, case reports, and expert 
committees). (See table 8.) 



Table 8: Categorization of Ratings Used by the Guideline Working Group 
Established by the Department of Veterans Affairs (VA)/Department of Defense 
(DOD) Evidence-Based Practice Work Group to Identify the Level of Evidence of 
Research Outcomes 

Rating Description 

I At least one properly done randomized controlled trial 

11-1 Well-designed controlled trial without randomization 

II-2 Well-designed cohort or case-control analytic study, preferably from more than 
one source 

II-3 Multiple time series evidence with/without intervention, dramatic results of 
uncontrolled experiment 

III Opinion of respected authorities, descriptive studies, case reports, and expert 
committees 

Source: VA/DOD document. 

Overall quality of research. After determining the level of evidence of 
individual research studies, the guideline working group makes a 
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determination regarding the overall quality of all of the research that 
addresses a particular clinical question. The overall quality takes into 
account the number, quality, and size of all of the individual research 
studies together as well as the consistency of the results between research 
outcomes to determine the collective overall strength of the research. 
Based on this review, the guideline working group determines the overall 
quality of the evidence to be good, fair, or poor. 9 (See table 9.) 



Table 9: Categorization of Ratings Used by the Guideline Working Group 
Established by the Department of Veterans Affairs (VA)/Department of Defense 
(DOD) Evidence-Based Practice Work Group to Determine the Overall Quality of 
Research 

Rating Description 

Good High-grade evidence (I or 11-1) linked to health outcome 

Fair High-grade evidence (l-or 11-1) linked to intermediate outcome 
Or 

Moderate-grade evidence (II-2 or II-3) directly linked to health outcome 
Poor Level III evidence or no linkage of evidence to health outcome 

Source: VA/DOD document. 

Net effect of the intervention. For interventions that were supported 
by studies of "fair" or "good" overall quality, the guideline working group 
evaluates the benefits and the potential harms to determine the net effect 
of the intervention. 10 The net effect of an intervention takes into account 
the benefits of the intervention minus the harms to determine the overall 
potential clinical benefit that the intervention may provide to patients. The 
net effect of the intervention ranges from "substantial" (meaning the 
benefit substantially outweighs the harm) to "zero or negative" (meaning it 
has no impact or a negative impact on patients). (See table 10.) 



"The overall quality of the research is considered "good" when the research outcomes are 
consistent from a number of higher-level research studies and have been conducted across 
a broad range of populations, thereby providing a high degree of confidence that the 
research outcomes are true. A "fair" overall quality indicates that the research outcomes 
could have been a result of true effects or biases present across some or all of the research 
studies. A "poor" rating for overall quality indicates that any conclusion about the research 
outcomes is uncertain because of serious methodological shortcomings, sparse data, or 
inconsistent results. 

10 The net effect of the intervention of research outcomes that are found to be of poor 
overall quality is not assessed because the evidence is insufficient to make a 
recommendation for or against routinely providing the intervention. 
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Table 10: Categorization of Ratings Used by the Guideline Working Group Established by the Department of Veterans Affairs 
(VA)/Department of Defense (DOD) Evidence-Based Practice Work Group to Determine the Net Effect of the Intervention as 
Described in the Research 



Rating Description 

Substantial More than a small relative impact on a frequent condition with a substantial burden of suffering 

Or 

A large impact on an infrequent condition with a significant impact on the individual patient level 

Moderate A small relative impact on a frequent condition with a substantial burden of suffering 

Or 

A moderate impact on an infrequent condition with a significant impact on the individual patient level 

Small A negligible relative impact on a frequent condition with a substantial burden of suffering 

Or 

A small impact on an infrequent condition with a significant impact on the individual patient level 

Zero or negative Negative impact on patients 
Or 

No relative impact on either a frequent condition with a substantial burden of suffering or an infrequent 
condition with a significant impact on the individual patient level 

Source: VA/DOD document. 

Grade of evidence-based recommendation. In the final step, the 
guideline working group uses its assessment of the overall quality of the 
evidence and the net effect of the intervention to grade evidence-based 
recommendations. (See table 11.) 



Table 1 1 : Grade of Evidence-Based Recommendation Assigned by the Guideline Working Group Established by the 


Department of Veterans Affairs (VA)/Department of Defense (DOD) Evidence-Based Practice Work Group Based on the 


Overall Quality of Evidence and the Net Effect of the Intervention as Described in the Research 




The net benefit of the intervention 


Overall quality of 




evidence 


Substantial Moderate Small Zero or negative 


Good 


A BCD 


Fair 


B BCD 


Poor I III 


Grade of 


Description of recommendation 


recommendation 




A 


A strong recommendation that the intervention is always indicated and acceptable 


B 


A recommendation that the intervention may be useful/effective 


C 


A recommendation that the intervention may be considered 


D 


A recommendation that a procedure may be considered not useful/effective or may be harmful 


I Insufficient evidence to recommend for or against — the clinician will use clinical judgment 



Source: VA/DOD document. 
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